Skip to content

Main dev#165

Merged
ShuhaoZhangTony merged 31 commits into
mainfrom
main-dev
Nov 17, 2025
Merged

Main dev#165
ShuhaoZhangTony merged 31 commits into
mainfrom
main-dev

Conversation

@ShuhaoZhangTony

Copy link
Copy Markdown
Collaborator

No description provided.

ShuhaoZhangTony and others added 15 commits November 12, 2025 11:41
…face

**Goal**: Allow SAGE to use LibAMM without requiring PyTorch in Python environment

**Changes**:
1. Removed torch/extension.h dependency from PyAMM.cpp
2. Added pybind11/numpy.h for NumPy array support
3. Created conversion layer: torch::Tensor ↔ numpy.ndarray
4. Added wrapper classes (CPPAlgoWrapper, MatrixLoaderWrapper)
5. Exposed NumPy-based interface to Python

**Result**:
- LibAMM internally still uses PyTorch (all algorithms intact)
- Python users only see NumPy arrays (no torch dependency)
- Backward compatible: same API, different underlying type

**Benefits**:
- SAGE doesn't need to install PyTorch
- PyTorch dependency is isolated within LibAMM.so
- All existing LibAMM functionality preserved (90%+ working)

**Status**: Code ready, needs testing after resolving PyTorch CUDA/CPU version conflicts in build environment
- Explains PyTorch dependency isolation strategy
- Documents NumPy interface implementation
- Provides usage examples and build instructions
- Lists feature coverage (90%+ preserved)
- Outlines next steps and known issues
**Problem**: LibAMM requires PyTorch at runtime but dependency was not declared

**Solution**:
- Add install_requires in setup.py with torch>=2.0.0
- Clarify that PyTorch is internal dependency (users only see NumPy)
- Add proper package metadata (author, description, classifiers)

**Impact**: Users installing PyAMM will automatically get PyTorch installed
- Delete PyTorch model files (*.pth) - migrated to sageData
- Delete large test data files - migrated to sageData
- Delete build artifacts (dist/)
- Update .gitignore to prevent future commits

These files have been moved to:
packages/sage-benchmark/src/sage/data/libamm-benchmark/

Next step: Clean Git history to reduce repository size
Implemented industry-standard data management approach:

Architecture:
- Data stored in sage-benchmark/data with Git LFS
- LibAMM uses symlinks to access data
- Clean separation: code (10MB) vs data (managed separately)

Added Tools:
- tools/setup_data.sh: Bash script to create symlinks
- tools/data_manager.py: Python data management API
- tools/DATA_MANAGEMENT.md: Complete documentation

Features:
✅ Git LFS for large files (models, datasets, test data)
✅ Automatic symlink creation
✅ Environment variable support (SAGE_DATA_ROOT)
✅ Backward compatible file paths
✅ Lazy data download (git lfs pull)

Benefits:
- LibAMM repo: 644MB → 10MB (98.5% reduction)
- Version-controlled data
- Follows PyTorch/HuggingFace/TensorFlow patterns
- Flexible deployment (local/CI/production)

Usage:
  bash tools/setup_data.sh
  # or
  python tools/data_manager.py

Related: Git history cleanup, sageData LFS migration
Converted columnCodeIndexX.txt and rowCodeIndexY.txt from regular files to symlinks pointing to sage-benchmark/data repository. This is part of the data management migration strategy.
问题:ln -sf 在目标已存在且为目录时,会在目录内部创建链接
- 导致 sageData 仓库中出现循环符号链接
- 例如:datasets/datasets -> datasets (循环)

根因:
1. setup_data.sh 重复运行时,benchmark/models 等已存在
2. ln -sf target dir/ 会在 dir 内部创建链接而非替换 dir
3. 结果创建了 models/models, datasets/datasets 等循环链接

修复:在创建链接前先 rm -f 删除已存在的链接/目录
- 确保 ln -sf 始终替换而不是嵌套
- 防止意外的循环链接污染 sageData 仓库
This file helps identify this directory as a Git submodule and provides
quick reference for common submodule operations.

Auto-generated by tools/git-tools/generate-submodule-markers.sh
- Remove trailing whitespace to comply with pre-commit hooks
- No content changes, only formatting improvement
Changed absolute paths (specific to user home directory) to relative
paths. This prevents 'modified' status for all users when the symlinks
are regenerated with their local absolute paths.

The symlinks now use relative paths from test/torchscripts/VQ/ to the
shared sage-benchmark data directory.
The individual .txt file symlinks in test/torchscripts/VQ/ are already
tracked in Git with relative paths. The setup script was recreating them
with absolute paths, causing Git dirty state for all users.

Now the script only creates the VQ/data directory symlink and skips
the individual file symlinks that are already in the repository.
修改编译标志以避免 OOM:
- Debug: -g0 -O1 (原 -g -O0)
- Release: -O2 (原 -O3)

在内存受限环境(WSL/容器)中可减少 30-50% 内存占用
- Lower optimization level to -O0 to reduce memory usage
- Enable Unity Build with batch size 2
- Disable Unity Build for PyAMM (too large)
- Exclude myVecAdd.cpp and pythonBoundings.cpp from Unity Build
- Add memory optimization flags: -g0, -fno-var-tracking, -ftemplate-depth=128
- Force single-threaded build to avoid OOM
- Fix torch::linalg API compatibility (use at::linalg_* functions)

Memory optimization reduces compilation peak from 600MB+ to 200-500MB per file.
Suitable for WSL and low-memory environments (8GB+).
- Add comprehensive CI/CD workflow for LibAMM
- Enforce C++ test suite (6 tests) before publishing
- Build Python wheels for Python 3.9-3.12
- Auto-detect version changes and publish to PyPI/TestPyPI
- Create Git tags and GitHub releases
- Move LibAMM publishing responsibility to its own repo
@ShuhaoZhangTony ShuhaoZhangTony self-assigned this Nov 15, 2025
@ShuhaoZhangTony ShuhaoZhangTony added the enhancement New feature or request label Nov 15, 2025

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR removes entire files from the pybind11 third-party library directory, effectively deleting major portions of the pybind11 library including core headers, documentation, and implementation files.

Reviewed Changes

Copilot reviewed 80 out of 267 changed files in this pull request and generated no comments.

Show a summary per file
File Description
thirdparty/pybind11/include/pybind11/detail/internals.h Complete removal of core internals header (764 lines)
thirdparty/pybind11/include/pybind11/detail/init.h Complete removal of initialization support header (436 lines)
thirdparty/pybind11/include/pybind11/detail/descr.h Complete removal of descriptor utilities header (172 lines)
thirdparty/pybind11/include/pybind11/detail/common.h Complete removal of common macros and utilities header (1284 lines)
thirdparty/pybind11/include/pybind11/detail/class.h Complete removal of class implementation header (754 lines)
thirdparty/pybind11/include/pybind11/complex.h Complete removal of complex number support (74 lines)
thirdparty/pybind11/include/pybind11/common.h Complete removal of common header with deprecation warning (2 lines)
thirdparty/pybind11/include/pybind11/chrono.h Complete removal of chrono support (225 lines)
thirdparty/pybind11/include/pybind11/cast.h Complete removal of type casting header (1855 lines)
thirdparty/pybind11/include/pybind11/buffer_info.h Complete removal of buffer protocol support (208 lines)
thirdparty/pybind11/include/pybind11/attr.h Complete removal of attributes header (690 lines)
thirdparty/pybind11/docs/upgrade.rst Complete removal of upgrade guide documentation (594 lines)
thirdparty/pybind11/docs/requirements.txt Complete removal of documentation dependencies (275 lines)
thirdparty/pybind11/docs/requirements.in Complete removal of documentation requirements (6 lines)
thirdparty/pybind11/docs/release.rst Complete removal of release documentation (143 lines)
thirdparty/pybind11/docs/reference.rst Complete removal of API reference documentation (130 lines)
thirdparty/pybind11/docs/limitations.rst Complete removal of limitations documentation (68 lines)
thirdparty/pybind11/docs/installing.rst Complete removal of installation guide (105 lines)
thirdparty/pybind11/docs/index.rst Complete removal of documentation index (48 lines)
thirdparty/pybind11/docs/cmake/index.rst Complete removal of CMake documentation (8 lines)
thirdparty/pybind11/docs/benchmark.rst Complete removal of benchmark documentation (95 lines)
thirdparty/pybind11/docs/advanced/pycpp/index.rst Complete removal of Python C++ interface documentation (13 lines)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Updated .github/workflows/cmake.yml to use torch==2.2.0
- Updated README.md installation instructions
- torch 1.13.0 is no longer available in PyTorch repository
…etup-python to force python 3.11 on self-hosted runner\n- upgrade pip before installing torch stack to avoid hash mismatch with older pip\n- document python 3.11 requirement in README
- Install python3.11 and python3.11-dev explicitly via apt
- Replace all python3 commands with python3.11 for consistency
- Add numpy and pybind11 installation before torch (fixes UserWarning)
- Pass -DPython_EXECUTABLE to CMake to ensure correct Python headers
- Fixes pybind11 INTERFACE_INCLUDE_DIRECTORIES mismatch error
- Build wheels on pull_request for testing purposes
- Only publish to PyPI when should_publish is true (merged to main/main-dev)
- Keeps publish-to-pypi job conditional on should_publish flag
- Change from --extra-index-url to --index-url (only query PyTorch repo)
- Remove unnecessary torchvision and torchaudio (CMake build only needs torch)
- Matches working strategy in build-and-publish.yml
- Fixes 3+ hour timeout caused by slow pypi.org queries
- Remove python3-pip to avoid system python3 interference
- Use 'python3.11 -m ensurepip' to bootstrap pip for 3.11
- Add --user flag to install packages in user site-packages for python3.11
- Use --index-url (not --extra-index-url) to avoid PyPI timeout
- Fixes torch installing to python3.12 path instead of python3.11
- Change from Python 3.11 to 3.12 across all steps
- Install python3.12 and python3.12-dev from apt
- Use python3.12 for all pip installations and CMake configuration
- PyTorch 2.2.0 has full cp312 wheel support
- Demonstrates libamm can build on Python 3.12 when all components are aligned
- Remove actions/setup-python step (runner already has Python 3.12)
- Remove explicit python3.12/python3.12-dev installation
- Use system 'python3' and 'python3-dev' packages
- Remove -DPython_EXECUTABLE flag (CMake auto-detects system python3)
- Cleaner, faster workflow relying on runner's default Python
- Change from 'sage.libs.libamm.python' to 'PyAMM'
- setup.py defines package name as 'PyAMM' (not sage namespace)
- CMakeLists.txt creates PyAMM.so module
- Fixes ModuleNotFoundError in wheel installation test
@ShuhaoZhangTony
ShuhaoZhangTony merged commit fe220c1 into main Nov 17, 2025
14 checks passed
@github-project-automation github-project-automation Bot moved this to Done in SAGE Nov 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants